Video enhancement using a neural network

ABSTRACT

Techniques for enhancing an image are described. For example, a lower-resolution image from a video file may be enhanced using a trained neural network applying the trained neural network on the lower-resolution image to remove artifacts by removing artifacts by generating, using a layer of the trained neural network, a residual value based on the proper subset of the received image and at least one corresponding image portion of a preceding lower resolution image in the video file and at least one corresponding image portion of a subsequent lower resolution image in the video file, upscale the lower-resolution image using bilinear upsampling, and combine the upscaled received image and residual value to generate an enhanced image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/403,355, filed May 3, 2019, which is hereby incorporated by referencein its entirety.

BACKGROUND

Streaming video is usually compressed to reduce bandwidth. The qualityof the compression and channel characteristics are determined by variousfactors including environmental factors and network congestion. Theseissues degrade the received image quality spatially and temporallyinducing artifacts.

BRIEF DESCRIPTION OF DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 is a diagram illustrating embodiments of an environment forenhancing images.

FIG. 2 is a diagram illustrating embodiments of an environment forenhancing images.

FIG. 3 illustrates embodiments of an image enhancement service or moduleto be used in inference.

FIG. 4 illustrates embodiments of an image enhancement service or moduleto be used in inference that uses a more recurrent approach.

FIG. 5 illustrates embodiments of an image enhancement service or moduleduring training.

FIG. 6 illustrates embodiments of a GAN of which part may be used togenerate enhanced images.

FIG. 7 illustrates embodiments of a generator with filters of a GAN ofwhich part may be used to generate enhanced images during training ofthe GAN.

FIG. 8 illustrates embodiments of a generator with filters ofprogressively trained GAN.

FIG. 9 illustrates embodiments of a generator with filters ofprogressively trained GAN.

FIG. 10 illustrates embodiments of a discriminator of a GAN.

FIG. 11 illustrates embodiments of an image enhancement service ormodule to be used in inference where the CNN to produce a higherresolution image is the generator of a GAN.

FIG. 12 is a flow diagram illustrating embodiments of a method forenhancing an image.

FIG. 13 a flow diagram illustrating embodiments of a method forenhancing an image.

FIG. 14 is a flow diagram illustrating embodiments of a method forenhancing an image.

FIG. 15 is a flow diagram illustrating embodiments of a method forenhancing an image.

FIG. 16 illustrates an example of a Pareto front.

FIG. 17 is a flow diagram illustrating a method for training a neuralnetwork using a Pareto front.

FIG. 18 illustrates some exemplary comparisons of various imagerenderings.

FIG. 19 illustrates an example provider network environment according tosome embodiments.

FIG. 20 illustrates an example data center that implements an overlaynetwork on a network substrate using IP tunneling technology accordingto some embodiments.

FIG. 21 is a block diagram of an example provider network that providesa storage service and a hardware virtualization service to customersaccording to some embodiments.

FIG. 22 is a block diagram illustrating an example computer system thatmay be used in some embodiments.

FIG. 23 illustrates a logical arrangement of a set of general componentsof an exemplary computing device that can be utilized in accordance withvarious embodiments.

FIG. 24 illustrates an example of an environment for implementingaspects in accordance with various embodiments.

DETAILED DESCRIPTION

Various embodiments of methods, apparatus, systems, and non-transitorycomputer-readable storage media for enhancing one or more images aredescribed. In particular, embodiments for removing compression artifactsand increasing video image resolution based on training convolutionalneural networks are detailed. These embodiments may operate on steamingvideo, stored video, or on static images, and may help with at least avariety of low-quality video problems such as: a) low quality video thathas occurred due to environmental circumstances; b) low quality videothat was purposefully created to reduce the bandwidth to save datatransfer; c) older videos that have been encoded with less quality orlower resolution; d) etc.

FIG. 1 is a diagram illustrating embodiments of an environment forenhancing images from still images or from a video. A provider network100 provides users with the ability to utilize one or more of a varietyof types of computing-related resources such as compute resources (e.g.,executing virtual machine (VM) instances and/or containers, executingbatch jobs, executing code without provisioning servers), data/storageresources (e.g., object storage, block-level storage, data archivalstorage, databases and database tables, etc.), network-related resources(e.g., configuring virtual networks including groups of computeresources, content delivery networks (CDNs), Domain Name Service (DNS)),application resources (e.g., databases, application build/deploymentservices), access policies or roles, identity policies or roles, machineimages, routers and other data processing resources, etc. These andother computing resources may be provided as services, such as ahardware virtualization service that can execute compute instances, astorage service that can store data objects, etc. The users (or“customers”) of provider networks 100 may utilize one or more useraccounts that are associated with a customer account, though these termsmay be used somewhat interchangeably depending upon the context of use.Users may interact with a provider network 100 across one or moreintermediate networks 106 (e.g., the internet) via one or moreinterface(s), such as through use of application programming interface(API) calls, via a console implemented as a website or application, etc.The interface(s) may be part of, or serve as a front-end to, a controlplane of the provider network 100 that includes “backend” servicessupporting and enabling the services that may be more directly offeredto customers.

To provide these and other computing resource services, providernetworks 100 often rely upon virtualization techniques. For example,virtualization technologies may be used to provide users the ability tocontrol or utilize compute instances (e.g., a VM using a guest operatingsystem (O/S) that operates using a hypervisor that may or may notfurther operate on top of an underlying host O/S, a container that mayor may not operate in a VM, an instance that can execute on “bare metal”hardware without an underlying hypervisor), where one or multiplecompute instances can be implemented using a single electronic device.Thus, a user may directly utilize a compute instance hosted by theprovider network to perform a variety of computing tasks or mayindirectly utilize a compute instance by submitting code to be executedby the provider network, which in turn utilizes a compute instance toexecute the code (typically without the user having any control of orknowledge of the underlying compute instance(s) involved).

For example, in various embodiments, a “serverless” function may includecode provided by a user or other entity that can be executed on demand.Serverless functions may be maintained within provider network 100 andmay be associated with a particular user or account, or may be generallyaccessible to multiple users and/or multiple accounts. Each serverlessfunction may be associated with a URL, URI, or other reference, whichmay be used to call the serverless function. Each serverless functionmay be executed by a compute instance, such as a virtual machine,container, etc., when triggered or invoked. In some embodiments, aserverless function can be invoked through an application programminginterface (“API”) call or a specially formatted HyperText TransportProtocol (“HTTP”) request message. Accordingly, users can defineserverless functions that can be executed on demand, without requiringthe user to maintain dedicated infrastructure to execute the serverlessfunction. Instead, the serverless functions can be executed on demandusing resources maintained by the provider network 100. In someembodiments, these resources may be maintained in a “ready” state (e.g.,having a pre-initialized runtime environment configured to execute theserverless functions), allowing the serverless functions to be executedin near real-time.

As shown in FIG. 1 , a customer can access an image enhancement serviceor module 108 in provider network 100 using a client device 102. Theclient device 102 can access the image enhancement service or module 108over one or more intermediate networks 106 through an interface providedby image enhancement service or module 108, such as an API, console,application, etc. In some embodiments, a user can upload one or moreimage files 111 to an input store 110 in a storage service 104. In someembodiments, the storage service may provide a virtual data store (e.g.,a folder or “bucket”, a virtualized volume, a database, etc.) providedby the provider network 100. The user may access the functionality ofstorage service 104, for example via one or more APIs, to store data tothe input store 110.

The image enhancement service or module 108 includes at least onetrained convolutional neural network (CNN) 112 and, in some embodiments,includes an object recognition module 114. The least one trained CNN 112performs image enhancement on at least a proper subset of an image aswill be detailed below. The object recognition module 114 finds one ormore particular objects (such as visage recognition) in an image. Insome embodiments, a CNN selector 113 uses an one or more of an inputimage (or a collection of them), a recognized object, the bandwidthavailable to the neural network (which may impact the initialresolution), processing power available, power, an acceptable latency,locality information for the image and/or destination viewer, memoryavailable, lighting information for the image, screen resolution, etc.select a trained CNN 112 to perform the image enhancement.

The circled numerals illustrate an exemplary flow of actions. At numeral1, the user sends a request to image enhancement service or module 108to enhance one or more images. The request may include at least one of alocation of the neural network (e.g., CNN 112) to use, a location of animage-based file (such as solo image file, a video file, etc.), adesired resolution, if object detection is to be used (and for what),etc. Additionally, a neural network 112 may have different profiles toutilize depending upon environmental factors such as processing power,power, etc.

At numeral 2, the image enhancement service or module 108 calls forimages 111 (such as for a video) according to the request. Note that theinput storage 110 may simply be a buffer and not a longer-term storage.

When object recognition is to be performed that occurs at numeral 3.Depending upon the implementation, the resultant proper subset of animage is provided to the trained CNN 112 at circle 5 or entire image isprovided, but a selected trained CNN 12 is used for that subset andpotentially a different trained CNN 112 is used for the rest of theimage.

The trained CNN 112 may access additional image content (such aspreceding and succeeding frames) at circle 6. The trained CNN 112performs image enhancement at circle 7 and provides the result(s) to theclient device(s) 102 at circle 8. Note, in some embodiments, an outputof the CNN 112 per image (or proper subset thereof) is multiple images(or proper subsets thereof). For example, an image for T₀ and T_(0.5)such that potentially a subsequent image in time may not need to begenerated “from scratch.”

FIG. 2 is a diagram illustrating embodiments of an environment forenhancing images. In this example, the environment is a client device102.

As shown in FIG. 2 , an application 201 can access an image enhancementservice or module 108. The image enhancement service or module 108includes at least one trained convolutional neural network (CNN) 112and, in some embodiments, includes an object recognition module 114. Theleast one trained CNN 112 performs image enhancement on at least aproper subset of an image of images 211 of storage 210 as will bedetailed below. The object recognition module 114 finds one or moreparticular objects (such as a visage) in an image. In some embodiments,a CNN selector 113 uses an one or more of an input image (or acollection of them), a recognized object, the bandwidth available to theneural network (which may impact the initial resolution), processingpower available, power, an acceptable latency, locality information forthe image and/or destination viewer, lighting information for the image,screen resolution, etc. select a trained CNN 112 to perform the imageenhancement.

The circled numerals illustrate an exemplary flow of actions. At numeral1, the application 201 sends a request to image enhancement service ormodule 108 to enhance one or more images. The request may include atleast one of a location of the neural network (e.g., CNN 112) to use, alocation of an image-based file (such as solo image file, a video file,etc.), a desired resolution, if object detection is to be used (and forwhat), etc. Additionally, a neural network 112 may have differentprofiles to utilize depending upon environmental factors such asprocessing power, power, etc.

At numeral 2, the image enhancement service or module 108 calls forimages 211 (such as for a video) according to the request. Note that thestorage 210 may simply be a buffer and not a longer-term storage.

When object recognition is to be performed that occurs at numeral 3.Depending upon the implementation, the resultant proper subset of animage is provided to the trained CNN 112 at circle 5 or entire image isprovided, but a selected trained CNN 12 is used for that subset andpotentially a different trained CNN 112 is used for the rest of theimage.

The trained CNN 112 may access additional image content (such aspreceding and succeeding frames) at circle 6. The trained CNN 112performs image enhancement at circle 7 and provides the result(s) to theclient device(s) 102 at circle 8.

FIG. 3 illustrates embodiments of an image enhancement service or moduleto be used in inference. The image enhancement service or module 108 isto be utilized to enhance one or more images, or proper subsets ofimages thereof, of a video in response to an inference request to do so.That image may be a part of a video stream or a stored video. In someembodiments, an image is a frame of video. Typically, although notnecessarily the case, the video is of a lower quality (lower resolution,etc.).

The image enhancement service or module 108 may be used in manydifferent environments. In some embodiments, the image enhancementservice or module 108 is a part of a provider network (e.g., as aservice) that can be called (such as shown in FIG. 1 ). In theseembodiments, the image enhancement service or module 108 receives avideo (streamed or stored), enhances some part of it as directed, andthen makes that enhancement available (such as by storing it ortransmitting it to a client device). In some embodiments, the imageenhancement service or module 108 is a part of a client device (such asshown in FIG. 2 ). In these embodiments, the image enhancement serviceor module 108 receives a video (streamed or stored), enhances some partof it as directed, and then makes that enhancement available (such as bymaking available for display or storing for later playback).

In some embodiments, an object recognition module 114 finds one or moreparticular objects (such as a visage) in an input image 305 to beenhanced.

In some embodiments, a CNN selector 113 uses one or more of an inputimage (or a collection of them), a recognized object, the bandwidthavailable to the neural network (which may impact the initialresolution), processing power available, power, an acceptable latency,locality information for the image and/or destination viewer, lightinginformation for the image, screen resolution, etc. select a trained CNN112 to perform the image enhancement.

In this illustration, the image enhancement service or module 108includes a convolutional neural network (CNN) 310 comprised of aplurality of layers 311 which may include one or more of: 1) one or moreconvolutional layers, 2) one or more subsampling layers, 3) one or morepooling layers, 4) and other layers. The CNN layers 311 operate one aninput image 305 (or a proper subset thereof) to be enhanced and one ormore preceding 301-303 and succeeding images 307-309 (or a proper subsetthereof in the video to be enhanced. Connected to these layers 311 is aCNN residual layer 313 that generates a residual value for the output ofthe CNN layers 313. The residual value removes or minimizes spatial andtemporal compression artifacts produced by the video encoder thatproduced the image such as blocking, ringing, and flickering. In someembodiments, an artifact removal layer (or layers) 317 removes artifactsfrom the input image 305.

In some embodiments, the image enhancement service or module 108 alsoincludes an image upsampler 315 (such as a bilinear upsampler) thatupsamples the image to be enhanced. In some embodiments, the image isupsampled four times, however, other upsampling scales may be used. Theupsampled image and the residual are summed to generate an enhancedimage.

In some embodiments, the image enhancement service or module 108includes an upsampling filter layer 316 in the CNN 310 that is coupledto the CNN layers 311 to upsample the image (or subset thereof). Atensor product of the upsampled image and the input image 305 isperformed and then summed with the residual to generate an enhancedimage.

FIG. 4 illustrates embodiments of an image enhancement service or moduleto be used in inference that uses a more recurrent approach. Inparticular, a previous higher resolution image will be an input into theCNN 310 instead of a plurality of preceding 301-303 and succeeding307-309 lower resolution images.

In some embodiments, for an initial image to enhance, the imageenhancement service or module 108 does utilize a preceding andsucceeding lower resolution images to generate a higher resolution imageas detailed above (such as using either the image upsampler 315 orupsampling filter layer 316). This higher resolution image is thenpassed back as an input to the CNN 310 along with the image to enhance.The CNN 310 only utilizes the single lower resolution image and apreviously generated higher resolution image in the generation of anenhanced image.

In other embodiments, the initial image to enhance is enhanced withoutany other input to the CNN 310, (e.g., the output of the CNN layers 311and any other layers needed to generate an enhanced image). The resultof that initial enhancement is then passed back in as an input to theCNN 310 along with the image to enhance The CNN 310 only utilizes thesingle lower resolution image and the previously generated higherresolution image in the generation of an enhanced image. This approachreduces a number of frames to process and may be faster than thepreviously detailed approach. In some embodiments, an artifact removallayer (or layers) 317 removes artifacts from the input image 305.

Note the artifact removal module 317, object recognition module 114, andCNN selector 113 operate as previously described.

Of course, prior to using a CNN for inference, it should be trained.FIG. 5 illustrates embodiments of an image enhancement service or moduleduring training. Many aspects are shared between the inference versionsdetailed above, however, in some embodiments, there is at least onedifference. In particular, the CNN layers 311 are shared by the CNNresidual layer 313 detailed above and CNN filters 316. The CNN filters316 perform the function of upsampling. During training, the CNN 310utilizes those filters 316 instead of a separate upsampling function.Further, during training the entire CNN 310 is trained including the CNNfilters 316 which are not used during inference (in some embodiments).

As streamed videos may lose sharpness and color vibrancy as signal isencoded and transmitted through the network, in some embodiments, anexaggerated target training images are used to promote the network toenhance sharpness and color on the output.

In some embodiments, the CNN 310 (or at least the CNN layers) is acomponent of a generative adversarial network (GAN). In particular, inthese embodiments the CNN 310 is a generator component of a GAN. In someembodiments, an artifact removal layer (or layers) 317 removes artifactsfrom the input image 305.

FIG. 6 illustrates embodiments of a GAN of which part may be used togenerate enhanced images. As shown, the GAN 600 includes a generatorwith filters 611 and a discriminator 613. The generator with filters 611are to produce a “fake” image from an image of a lower resolution imagedataset 601 and the discriminator 613 is to compare the “fake” image toa corresponding higher resolution image from a higher resolution dataset603 to determine if the image is “real” or “fake” (evaluates whether thegenerate image belongs in the training data set of higher resolutionimages or not). The output of the discriminator 613 is a probabilitythat the generated image is fake. The discriminator 613 is a fullyconnected neural network. If the generator 611 is performing well(generating good “fakes”) then the discriminator 613 will return a valueindicating a higher probability of the generated images being real.

The constructed image of the generator with filters 611 is alsosubtracted from a corresponding image of the higher resolution imagedataset 601 which indicates a perceptual loss. This perceptual loss isadded to the output of the discriminator 613 to produce a generator losswhich is fed back to the generator with filters 611 to update theweights of the generator with filters 611 to help train it. Adiscriminator loss is the inverse of the generator loss and is fed backto the discriminator 613 to update its weights. Note that in someembodiments the discriminator 613 and generator with filters 611 aretrained in turns by first training one and then the other.

As streamed videos may lose sharpness and color vibrancy as signal isencoded and transmitted through the network, in some embodiments, anexaggerated target training image is used as the higher resolution imageto promotes the network to enhance sharpness and color on the output.

In some embodiments, the output of the generator with filters 611 isadded to an upsampled input image 305. In some embodiments, one or moreof an object recognition module 114 and a CNN selector 113 are utilizedand operate as previously described.

FIG. 7 illustrates embodiments of a generator with filters of a GAN ofwhich part may be used to generate enhanced images during training ofthe GAN. As noted above, in some embodiments the discriminator 613 andgenerator with filters 611 are trained progressively. In progressivetraining, both the generator with filters 611 and discriminator 613start off at lower resolution and new layers that produce increasinglyhigher resolutions are added during training When a new set of layersare added, they are slowly blended in over several epochs and thenstabilized for several more epochs.

The generator with filters 611 and discriminator 613 comprise multiplestages each and this illustration is for embodiments of the generatorwith filters 611.

In some embodiments, the generator with filters 611 performs sometemporal pre-processing (such as concatenating lower resolution imagesalong the temporal dimension) using a temporal pre-processing layer(s)703. The temporal dimension is then reduced from 2n+1 to 1 using atemporal reduction layer(s) 705 which, in some embodiments, consists ofa set of 3×3×3 3D convolutions with no padding in the temporaldimension.

In this particular example (having a 4× upscaling), the generator withfilters 611 are trained in multiple stages. In a first stage, thegenerator with filters 611 is trained to only remove compressionartifacts and output an image that has the same resolution (1×) as theinput images and features using artifact removal layer(s) 707. In thisstage, the generator with filters 611 are initialized using only meansquared error (MSE) and perceptual losses. Note that because thegenerator with filters 611 are trained to output an image at the 1×stage by enforcing perceptual and MSE losses throughout all trainingstages, the 1× value can be used to verify the model's ability to removeencoding artifacts at this stage.

A second stage (comprising 2× layer(s) 709, upsampling layer(s) 711, and2× filters 717) processes the features of the 1× image (using the 2×layer(s) 709) to generate a higher resolution image (using the 2×layer(s) 709) and upsample features of the higher resolution image usingthe upsampling layer(s) 711. Note the discriminator 613 may also beenabled. In some embodiments, the 2× layer(s) 709 is/are one or more CNNlayers. The adversarial loss is used in addition to MSE and perceptuallosses for training the generator with filters 611. In some embodiments,the 2× filter(s) 717 are generated from the output features of theartifact removal layer(s) 707. This allows the generator with filters611 to learn to upsample the image after artifacts were removed. Notethat for efficiency, in some embodiments, most of the computation isperformed in 1× resolution and only at the end a depth-to-space layerfor 2× upsampling is performed using 2× upsampling layer(s) 711. The 2×layer(s) 709 for both the generator with filters 611 and thediscriminator 613 are blended over several epochs and then stabilizedfor several more epochs. The product of the output of the 2× filters 717and the 1× RGB image is added to the output of the 2× upsamplinglayer(s) 711 to produce the 2× RGB image(s) which the discriminator 613consumes.

A third stage is used for blending and stabilizing the 4× output in ananalogous way to the second stage using 4× filters 719, 4× layer(s) 713on the output of the 2× upsampling layer(s) 711, and 4× upsamplinglayer(s) 715. After the 4× layers have been blended in completely, thelosses on the 2× output are no longer enforced. 4× RGB images areproduced using a product of the output of the 4× filter(s) 719 with the1× RGB image(s) produced by the artifact removal layer(s) 707 that hasbeen summed with the output of the 4× upsampling layer(s) 715.

FIG. 8 illustrates embodiments of a generator with filters 611 ofprogressively trained GAN. As shown, the filtering procedure usingfilter(s) 819 is simplified from training in that the filter(s) 819 onlyact on the output of the artifact removal layer(s) 707.

FIG. 9 illustrates embodiments of a generator with filters 611 ofprogressively trained GAN. This is a more simplified version of thegenerator with filters 611 of FIG. 8 where the 2× layer(s) 709, 2×upsampling layer(s) 711, and 4× layer(s) 713 are not utilized.

FIG. 10 illustrates embodiments of a discriminator of a GAN. This GANtakes in the RGB values from the generator with filters 711. The RGB 4×values are subjected to a 2D convolution 1001 using 64 channels followedby a leaky ReLU (LReLU) 1003. The output of the LReLU is added to theRGB 2× values and then subjected to a 2D convolution using 128 channelsfollowed by batch normalization 1005 and another LReLU.

The output of the second LReLU is added to the RGB 1× values and thensubjected to a plurality of convolution-BN-LReLU combinations using 2Dconvolutions from 256 to 2048 to 256 channels.

One or more dense layers are used to flatten the result and then asigmoid activation function 1011 is applied.

FIG. 11 illustrates embodiments of an image enhancement service ormodule to be used in inference where the CNN to produce a higherresolution image is the generator of a GAN. The image enhancementservice or module 108 is to be utilized to enhance one or more images,or proper subsets of images thereof, of a video in response to aninference request to do so as noted previously. Again, note what is notshown in this illustration is any feature detection means which may beutilized to set what aspects of an image are to be enhanced (such as avisage). In this illustration, the image enhancement service or module108 includes a convolutional neural network (CNN) that is the generatorwith filters 611 of a GAN. A lower resolution image 305 and/or one ormore surrounding images 300 is/are passed to the generator with filters611 which produces the higher resolution image.

FIG. 12 is a flow diagram illustrating embodiments of a method forenhancing an image. Depending upon the implementation, this method maybe performed as a part of a service of a provider network, on a clientdevice (sender and/or receiver), and/or a combination thereof.

In some embodiments, at 1201, a neural network to enhance an image, or aproper subset thereof, is trained. For example, a generator or generatorwith filters of a GAN or one or more CNN layers are trained as discussedherein. Note that a neural network may be trained for differentenvironments. For example, a different neural network may be useddepending upon the bandwidth available to the neural network (which mayimpact the initial resolution), processing power available, poweravailable (e.g., it may not be best to render very high-resolutionimages on a mobile device), an acceptable latency, etc.

At 1202, a request to enhance one or more images is received. Thisrequest may include at least one of a location of the neural network touse, a location of an image-based file (such as solo image file, a videofile, etc.), a desired resolution, which images to enhance (for example,not enhance higher quality frames), etc. Additionally, a stored neuralnetwork may have different profiles to utilize depending uponenvironmental factors such as processing power, power, etc. The requestmay come from a client device user, an application, etc.

An image to be at least partially enhanced using a trained neuralnetwork is received at 1203. For example, a lower resolution image thatis a part of a video stream or file is received by an image enhancementservice or module which have been detailed above.

In some embodiments, a determination of if the received image should beenhanced is made at 1204. In some embodiments, as some images arrivewith better quality than others, the higher quality frames may be foundby an image quality assessment or by appending a known pattern to everyframe from where to judge the distortion. Depending upon the processingcapabilities, available bandwidth, etc. the different types of imagesmay be handled differently. For example, higher quality images may notbe enhanced by either a sender or a receiver. Assessing image qualityper image to potentially enhance may also be used purposefully to reducebandwidth not enhancing some frames with already higher quality andusing those higher quality frames to help get a better overall output ata reduced average bitrate. If not, then the next image to potentiallyenhanced is received at 1203.

If so, then in some embodiments, a determination of a proper subset ofthe received image to enhance is made at 1205. For example, an objectrecognition of a visage, etc. is made. Streamed and compressed imagescontain some regions that are more informative than others or morerelevant such as people, edges of objects, center of the image, etc.Operating on fixing the entire image may be too expensivecomputationally or unnecessary. Note that in some embodiments, a senderusing this method may reduce bandwidth by sending the most relevantparts of the image with less compression than the remainder of theimage. In some embodiments, when this method is performed by a sender,the encodes information with the image itself that notes the image, or aproper subset thereof, is important so that the receiving end onlyapplies the removal of artifacts on pre-defined areas making the processfaster and also requiring less bandwidth overall. In those embodiments,the receiver can then determine what to enhance in an image.

In some embodiments, a determination of one or more CNNs to use forenhancement is made at 1206. Examples of what goes into thatdetermination have been detailed.

The (proper subset of the) received image is enhanced according to therequest at 1207. The trained neural network generates a residual valuebased on the (proper subset of the) received image and at least onecorresponding image portion of a preceding lower resolution image and atleast one corresponding image portion of a subsequent lower resolutionimage at 1209. The (proper subset of the) received image is upscaled at1211. The upscaled (proper subset of the) received image and residualvalue are added to generate an enhanced image of the (proper subset ofthe) received image at 1213. Note that multiple CNNs may be used on asingle image. For example, a proper subset of an image may use one CNNand the rest of the image may use a different CNN, etc.

The enhanced image is output at 1215. For example, the enhanced image isstored, displayed, etc.

In some embodiments, the enhanced image is merged with other(potentially unenhanced) images of a video file to generate a higherquality video file at 1217. For example, a device with very low memorycould store very low resolution video locally, mix this local data withenhanced streamed low bitrate video to create a better quality output.

FIG. 13 is a flow diagram illustrating embodiments of a method forenhancing an image. Depending upon the implementation, this method maybe performed as a part of a service of a provider network, on a clientdevice (sender and/or receiver), and/or a combination thereof.

In some embodiments, at 1301, a neural network to enhance an image, or aproper subset thereof, is trained. For example, a generator or generatorwith filters of a GAN or one or more CNN layers are trained as discussedherein. Note that a neural network may be trained for differentenvironments. For example, a different neural network may be useddepending upon the bandwidth available to the neural network (which mayimpact the initial resolution), processing power available, poweravailable (e.g., it may not be best to render very high-resolutionimages on a mobile device), an acceptable latency, etc.

At 1302, a request to enhance one or more images is received. Thisrequest may include at least one of a location of the neural network touse, a location of an image-based file (such as solo image file, a videofile, etc.), a desired resolution, which images to enhance (for example,not enhance higher quality frames), etc. Additionally, a stored neuralnetwork may have different profiles to utilize depending uponenvironmental factors such as processing power, power, etc. The requestmay come from a client device user, an application, etc.

An image to be at least partially enhanced using a trained neuralnetwork is received at 1303. For example, a lower resolution image thatis a part of a video stream or file is received by an image enhancementservice or module which have been detailed above.

In some embodiments, a determination of if the received image should beenhanced is made at 1304. In some embodiments, as some images arrivewith better quality than others, the higher quality frames may be foundby an image quality assessment or by appending a known pattern to everyframe from where to judge the distortion. Depending upon the processingcapabilities, available bandwidth, etc. the different types of imagesmay be handled differently. For example, higher quality images may notbe enhanced by either a sender or a receiver. Assessing image qualityper image to potentially enhance may also be used purposefully to reducebandwidth not enhancing some frames with already higher quality andusing those higher quality frames to help get a better overall output ata reduced average bitrate. If not, then the next image to potentiallyenhanced is received at 1303.

If so, then in some embodiments, a determination of a proper subset ofthe received image to enhance is made at 1305. For example, an objectrecognition of a visage, etc. is made. Streamed and compressed imagescontain some regions that are more informative than others or morerelevant such as people, edges of objects, center of the image, etc.Operating on fixing the entire image may be too expensivecomputationally or unnecessary. Note that in some embodiments, a senderusing this method may reduce bandwidth by sending the most relevantparts of the image with less compression than the remainder of theimage. In some embodiments, when this method is performed by a sender,the encodes information with the image itself that notes the image, or aproper subset thereof, is important so that the receiving end onlyapplies the removal of artifacts on pre-defined areas making the processfaster and also requiring less bandwidth overall. In those embodiments,the receiver can then determine what to enhance in an image.

In some embodiments, a determination of a CNN to use for enhancement ismade at 1306. Examples of what goes into that determination have beendetailed.

The (proper subset of the) received image is enhanced at 1307 using thetrained neural network to generate an enhanced image of the (propersubset of the) received image based on the (proper subset of the)received image and a previously generated higher resolution image. Insome embodiments, the previously generated higher resolution image wasgenerated according to 807. In other embodiments, the previouslygenerated higher resolution image was simply made by using the trainedneural network without having a second input. Note that multiple CNNsmay be used on a single image. For example, a proper subset of an imagemay use one CNN and the rest of the image may use a different CNN, etc.

The enhanced image is output at 1309. For example, the enhanced image isstored, displayed, etc.

In some embodiments, the enhanced image is merged with other(potentially unenhanced) images of a video file to generate a higherquality video file at 1311. For example, a device with very low memorycould store very low resolution video locally, mix this local data withenhanced streamed low bitrate video to create a better quality output.

FIG. 14 is a flow diagram illustrating embodiments of a method forenhancing an image. Depending upon the implementation, this method maybe performed as a part of a service of a provider network, on a clientdevice (sender and/or receiver), and/or a combination thereof.

In some embodiments, at 1401, a neural network to enhance an image, or aproper subset thereof, is trained. For example, a generator or generatorwith filters of a GAN or one or more CNN layers are trained as discussedherein. Note that a neural network may be trained for differentenvironments. For example, a different neural network may be useddepending upon the bandwidth available to the neural network (which mayimpact the initial resolution), processing power available, poweravailable (e.g., it may not be best to render very high-resolutionimages on a mobile device), an acceptable latency, locality (forexample, for a particular geography/look a network can be trained thatworks well with images from a particular location, lighting (day, night,inside, etc.), screen resolution, etc.

At 1402, a request to enhance one or more images is received. Thisrequest may include at least one of a location of the neural network touse, a location of an image-based file (such as solo image file, a videofile, etc.), a desired resolution, etc. Additionally, a stored neuralnetwork may have different profiles to utilize depending uponenvironmental factors such as processing power, power, etc. The requestmay come from a client device user, an application, etc.

An image to be at least partially enhanced using a trained neuralnetwork is received at 1403. For example, a lower resolution image thatis a part of a video stream or file is received by an image enhancementservice or module which have been detailed above.

In some embodiments, a determination of if the received image should beenhanced is made at 1404. In some embodiments, as some images arrivewith better quality than others, the higher quality frames may be foundby an image quality assessment or by appending a known pattern to everyframe from where to judge the distortion. Depending upon the processingcapabilities, available bandwidth, etc. the different types of imagesmay be handled differently. For example, higher quality images may notbe enhanced by either a sender or a receiver. Assessing image qualityper image to potentially enhance may also be used purposefully to reducebandwidth not enhancing some frames with already higher quality andusing those higher quality frames to help get a better overall output ata reduced average bitrate. If not, then the next image to potentiallyenhanced is received at 1403.

If so, then in some embodiments, a determination of a proper subset ofthe received image to enhance is made at 1405. For example, an objectrecognition of a visage, etc. is made. Streamed and compressed imagescontain some regions that are more informative than others or morerelevant such as people, edges of objects, center of the image, etc.Operating on fixing the entire image may be too expensivecomputationally or unnecessary. Note that in some embodiments, a senderusing this method may reduce bandwidth by sending the most relevantparts of the image with less compression than the remainder of theimage. In some embodiments, when this method is performed by a sender,the encodes information with the image itself that notes the image, or aproper subset thereof, is important so that the receiving end onlyapplies the removal of artifacts on pre-defined areas making the processfaster and also requiring less bandwidth overall. In those embodiments,the receiver can then determine what to enhance in an image.

The (proper subset of the) received image is enhanced according to therequest at 1407. The trained neural network generates a residual valuebased on the (proper subset of the) received image and a previouslygenerated higher resolution image. The (proper subset of the) receivedimage is upscaled at 1411. The upscaled (proper subset of the) receivedimage and residual value are added to generate an enhanced image of the(proper subset of the) received image at 1413. Note that multiple CNNsmay be used on a single image. For example, a proper subset of an imagemay use one CNN and the rest of the image may use a different CNN, etc.

The enhanced image is output at 1415. For example, the enhanced image isstored, displayed, etc.

In some embodiments, the enhanced image is merged with other(potentially unenhanced) images of a video file to generate a higherquality video file at 1417. For example, a device with very low memorycould store very low resolution video locally, mix this local data withenhanced streamed low bitrate video to create a better quality output.

FIG. 15 is a flow diagram illustrating embodiments of a method forenhancing an image. Depending upon the implementation, this method maybe performed as a part of a service of a provider network, on a clientdevice (sender and/or receiver), and/or a combination thereof.

In some embodiments, at 1501, a neural network to enhance an image, or aproper subset thereof, is trained. For example, a generator or generatorwith filters of a GAN or one or more CNN layers are trained as discussedherein. Note that a neural network may be trained for differentenvironments. For example, a different neural network may be useddepending upon the bandwidth available to the neural network (which mayimpact the initial resolution), processing power available, poweravailable (e.g., it may not be best to render very high-resolutionimages on a mobile device), an acceptable latency, locality (forexample, for a particular geography/look a network can be trained thatworks well with images from a particular location, lighting (day, night,inside, etc.), screen resolution, etc.

At 1502, a request to enhance one or more images is received. Thisrequest may include at least one of a location of the neural network touse, a location of an image-based file (such as solo image file, a videofile, etc.), a desired resolution, etc. Additionally, a stored neuralnetwork may have different profiles to utilize depending uponenvironmental factors such as processing power, power, etc. The requestmay come from a client device user, an application, etc.

An image to be at least partially enhanced using a trained neuralnetwork is received at 1503. For example, a lower resolution image thatis a part of a video stream or file is received by an image enhancementservice or module which have been detailed above.

In some embodiments, a determination of if the received image should beenhanced is made at 1504. In some embodiments, as some images arrivewith better quality than others, the higher quality frames may be foundby an image quality assessment or by appending a known pattern to everyframe from where to judge the distortion. Depending upon the processingcapabilities, available bandwidth, etc. the different types of imagesmay be handled differently. For example, higher quality images may notbe enhanced by either a sender or a receiver. Assessing image qualityper image to potentially enhance may also be used purposefully to reducebandwidth not enhancing some frames with already higher quality andusing those higher quality frames to help get a better overall output ata reduced average bitrate. If not, then the next image to potentiallyenhanced is received at 1503.

If so, then in some embodiments, a determination of a proper subset ofthe received image to enhance is made at 1505. For example, an objectrecognition of a visage, etc. is made. Streamed and compressed imagescontain some regions that are more informative than others or morerelevant such as people, edges of objects, center of the image, etc.Operating on fixing the entire image may be too expensivecomputationally or unnecessary. Note that in some embodiments, a senderusing this method may reduce bandwidth by sending the most relevantparts of the image with less compression than the remainder of theimage. In some embodiments, when this method is performed by a sender,the encodes information with the image itself that notes the image, or aproper subset thereof, is important so that the receiving end onlyapplies the removal of artifacts on pre-defined areas making the processfaster and also requiring less bandwidth overall. In those embodiments,the receiver can then determine what to enhance in an image.

The (proper subset of the) received image is enhanced using a generatorwith filters according to the request at 1507 as follows. Thelower-resolution image and one or more neighboring images arepre-processed by concatenating the lower-resolution images along atemporal dimension. This is followed, in some embodiments, with atemporal reduction of the concatenated images. Artifacts are the(temporally reduced) concatenated images are removed at a firstresolution to generate a first red, green, blue (RGB) image using anartifact removal layer and features of the first RGB image.

In some embodiments, features of the first RGB image are processed at asecond, higher resolution to generate a second RGB image and features ofthat second, higher resolution image are upsampled. The features of thesecond RGB image are then processed at a third, higher resolution togenerate a third RGB image and the features of the third RGB image areupsampled to generate a residual of the third RGB image. In someembodiments, a filter from the features of the first RGB image isgenerated and a product of the generated filter and the RGB imagegenerated by the artifact removal layer is performed. A sum of theproduct with the residual of the third RGB image generates an enhancedimage.

In some embodiments, features output from the artifact removal layer areupsampled to generate a residual. A sum of the residual with an inputRGB image generates an enhanced image.

The enhanced image is output at 1515. For example, the enhanced image isstored, displayed, etc.

In some embodiments, the enhanced image is merged with other(potentially unenhanced) images of a video file to generate a higherquality video file at 1517. For example, a device with very low memorycould store very low resolution video locally, mix this local data withenhanced streamed low bitrate video to create a better quality output.

Typically, the design of CNNs (such as those detailed above) is mostlyhand-crafted. This can be inefficient. Detailed here are embodiments ofa method to help automatically generate and quickly assess potentialnetworks via an optimization algorithm. In particular, elements such asnumber of layers, filter sizes, time windows, and others are(hyper)parameterized. For a given proposed network, parameters areencoded with a gradient that lead to them. That is for those CNNs thatare more desirable it is possible to know which direction (parameter)helped them.

Within streaming video enhancement, there are generally two maincriteria: image quality and speed of processing per frame. Image qualityis difficult to measure quantitatively and most commonly done viametrics such as peak signal to noise ratio (PSNR). This and othersimilar metrics are only roughly indicative of true perceptual qualitybut enough to guide the optimization algorithm.

To keep track of the best overall CNNs in this dual criteria space, aPareto front is used. FIG. 16 illustrates an example of a createdsamples for a Pareto front The Pareto front is a graph of solutions thatare better in one or more aspects over other solutions and allows forhundreds of networks to be assessed and to perform optimization on asmaller subset (the front) for further iterations and to select thecurrent best performers from that. As shown, a first network 1601 has aplurality of parameters and a gradient with respect to a parent network.The hashed boxes represent a Pareto (optimal) value after evaluating thefirst network 1601. In this example, the F parameter is optimal with avalue of 8.

The parameters of the first network 1601 are added to the gradient and a(random) mutation 1603 is introduced to form a child network 1605 toanalyze. After the analysis, another Pareto value (this time forparameter C) has been found. More mutations can be applied to either thefirst network 1601 or the child network 1605 to figure out Pareto valuesfor each of the parameters.

FIG. 17 is a flow diagram illustrating a method for training a neuralnetwork using a Pareto front. At 1701, a model is evaluated to measureperformance (accuracy and time) for the model having a set ofhyperparameters and determine gradient values for those hyperparameterswith respect to a parent model. In some embodiments, a plot of the modelis made such as plotting peak signal-to-noise ratio (PSNR) betweenimages and speed.

Pareto hyperparameters are tracked that provide desired performance forthe model with respect to a particular hyperparameter at 1703.

The gradient values are added to the non-Pareto hyperparameters andintroduce a random mutation to at least some of those non-Paretohyperparameters at 1705.

At 1707, the model with the changes is evaluated to measure performance(accuracy and time) for the model using the Pareto hyperparameters, themutated non-pareto parameters, and remaining hyperparameters.

The cycle of tracking Pareto hyperparameters, etc. may continue untilall of the Pareto hyperparameters have been found or until the processis stopped for another reason.

FIG. 18 illustrates some exemplary comparisons of various imagerenderings. Non-enhanced images 1801 show a noticeable amount of blur.CNN enhanced images 1803 show a marked improvement from that. A furtherimprovement of using a generator from a GAN for the CNN is shown inimages 1805.

FIG. 19 illustrates an example provider network (or “service providersystem”) environment according to some embodiments. A provider network1900 may provide resource virtualization to customers via one or morevirtualization services 1910 that allow customers to purchase, rent, orotherwise obtain instances 1912 of virtualized resources, including butnot limited to computation and storage resources, implemented on deviceswithin the provider network or networks in one or more data centers.Local Internet Protocol (IP) addresses 1916 may be associated with theresource instances 1912; the local IP addresses are the internal networkaddresses of the resource instances 1912 on the provider network 1900.In some embodiments, the provider network 1900 may also provide publicIP addresses 1914 and/or public IP address ranges (e.g., InternetProtocol version 4 (IPv4) or Internet Protocol version 6 (IPv6)addresses) that customers may obtain from the provider 1900.

Conventionally, the provider network 1900, via the virtualizationservices 1910, may allow a customer of the service provider (e.g., acustomer that operates one or more client networks 1950A-1950C includingone or more customer device(s) 1952) to dynamically associate at leastsome public IP addresses 1914 assigned or allocated to the customer withparticular resource instances 1912 assigned to the customer. Theprovider network 1900 may also allow the customer to remap a public IPaddress 1914, previously mapped to one virtualized computing resourceinstance 1912 allocated to the customer, to another virtualizedcomputing resource instance 1912 that is also allocated to the customer.Using the virtualized computing resource instances 1912 and public IPaddresses 1914 provided by the service provider, a customer of theservice provider such as the operator of customer network(s) 1950A-1950Cmay, for example, implement customer-specific applications and presentthe customer's applications on an intermediate network 1940, such as theInternet. Other network entities 1920 on the intermediate network 1940may then generate traffic to a destination public IP address 1914published by the customer network(s) 1950A-1950C; the traffic is routedto the service provider data center, and at the data center is routed,via a network substrate, to the local IP address 1916 of the virtualizedcomputing resource instance 1912 currently mapped to the destinationpublic IP address 1914. Similarly, response traffic from the virtualizedcomputing resource instance 1912 may be routed via the network substrateback onto the intermediate network 1940 to the source entity 1920.

Local IP addresses, as used herein, refer to the internal or “private”network addresses, for example, of resource instances in a providernetwork. Local IP addresses can be within address blocks reserved byInternet Engineering Task Force (IETF) Request for Comments (RFC) 1918and/or of an address format specified by IETF RFC 4193, and may bemutable within the provider network. Network traffic originating outsidethe provider network is not directly routed to local IP addresses;instead, the traffic uses public IP addresses that are mapped to thelocal IP addresses of the resource instances. The provider network mayinclude networking devices or appliances that provide network addresstranslation (NAT) or similar functionality to perform the mapping frompublic IP addresses to local IP addresses and vice versa.

Public IP addresses are Internet mutable network addresses that areassigned to resource instances, either by the service provider or by thecustomer. Traffic routed to a public IP address is translated, forexample via 1:1 NAT, and forwarded to the respective local IP address ofa resource instance.

Some public IP addresses may be assigned by the provider networkinfrastructure to particular resource instances; these public IPaddresses may be referred to as standard public IP addresses, or simplystandard IP addresses. In some embodiments, the mapping of a standard IPaddress to a local IP address of a resource instance is the defaultlaunch configuration for all resource instance types.

At least some public IP addresses may be allocated to or obtained bycustomers of the provider network 1900; a customer may then assign theirallocated public IP addresses to particular resource instances allocatedto the customer. These public IP addresses may be referred to ascustomer public IP addresses, or simply customer IP addresses. Insteadof being assigned by the provider network 1900 to resource instances asin the case of standard IP addresses, customer IP addresses may beassigned to resource instances by the customers, for example via an APIprovided by the service provider. Unlike standard IP addresses, customerIP addresses are allocated to customer accounts and can be remapped toother resource instances by the respective customers as necessary ordesired. A customer IP address is associated with a customer's account,not a particular resource instance, and the customer controls that IPaddress until the customer chooses to release it. Unlike conventionalstatic IP addresses, customer IP addresses allow the customer to maskresource instance or availability zone failures by remapping thecustomer's public IP addresses to any resource instance associated withthe customer's account. The customer IP addresses, for example, enable acustomer to engineer around problems with the customer's resourceinstances or software by remapping customer IP addresses to replacementresource instances.

FIG. 20 illustrates an example data center that implements an overlaynetwork on a network substrate using IP tunneling technology, accordingto some embodiments. A provider data center 2000 may include a networksubstrate that includes networking nodes 2012 such as routers, switches,network address translators (NATs), and so on, which may be implementedas software, hardware, or as a combination thereof. Some embodiments mayemploy an Internet Protocol (IP) tunneling technology to provide anoverlay network via which encapsulated packets may be passed throughnetwork substrate 2010 using tunnels. The IP tunneling technology mayprovide a mapping and encapsulating system for creating an overlaynetwork on a network (e.g., a local network in data center 2000 of FIG.20 ) and may provide a separate namespace for the overlay layer (thepublic IP addresses) and the network substrate 2010 layer (the local IPaddresses). Packets in the overlay layer may be checked against amapping directory (e.g., provided by mapping service 2030) to determinewhat their tunnel substrate target (local IP address) should be. The IPtunneling technology provides a virtual network topology (the overlaynetwork); the interfaces (e.g., service APIs) that are presented tocustomers are attached to the overlay network so that when a customerprovides an IP address to which the customer wants to send packets, theIP address is run in virtual space by communicating with a mappingservice (e.g., mapping service 2030) that knows where the IP overlayaddresses are.

In some embodiments, the IP tunneling technology may map IP overlayaddresses (public IP addresses) to substrate IP addresses (local IPaddresses), encapsulate the packets in a tunnel between the twonamespaces, and deliver the packet to the correct endpoint via thetunnel, where the encapsulation is stripped from the packet. In FIG. 20, an example overlay network tunnel 2034A from a virtual machine (VM)2024A (of VMs 2024A1-2024A4, via VMM 2022A) on host 2020A to a device onthe intermediate network 2050 and an example overlay network tunnel2034B between a VM 2024A (of VMs 2024A1-2024A4, via VMM 2022A) on host2020A and a VM 2024B (of VMs 2024B1-2024B4, via VMM 2022B) on host 2020Bare shown. In some embodiments, a packet may be encapsulated in anoverlay network packet format before sending, and the overlay networkpacket may be stripped after receiving. In other embodiments, instead ofencapsulating packets in overlay network packets, an overlay networkaddress (public IP address) may be embedded in a substrate address(local IP address) of a packet before sending, and stripped from thepacket address upon receiving. As an example, the overlay network may beimplemented using 32-bit IPv4 (Internet Protocol version 4) addresses asthe public IP addresses, and the IPv4 addresses may be embedded as partof 128-bit IPv6 (Internet Protocol version 6) addresses used on thesubstrate network as the local IP addresses.

Referring to FIG. 20 , at least some networks in which embodiments maybe implemented may include hardware virtualization technology thatenables multiple operating systems to run concurrently on a hostcomputer (e.g., hosts 2020A and 2020B of FIG. 20 ), i.e. as virtualmachines (VMs) 2024 on the hosts 2020. The VMs 2024 may, for example, beexecuted in slots on the hosts 2020 that are rented or leased tocustomers of a network provider. A hypervisor, or virtual machinemonitor (VMM) 2022, on a host 2020 presents the VMs 2024 on the hostwith a virtual platform and monitors the execution of the VMs 2024. EachVM 2024 may be provided with one or more local IP addresses; the VMM2022 on a host 2020 may be aware of the local IP addresses of the VMs2024 on the host. A mapping service 2030 may be aware of (e.g., viastored mapping information 2032) network IP prefixes and IP addresses ofrouters or other devices serving IP addresses on the local network. Thisincludes the IP addresses of the VMMs 2022 serving multiple VMs 2024.The mapping service 2030 may be centralized, for example on a serversystem, or alternatively may be distributed among two or more serversystems or other devices on the network. A network may, for example, usethe mapping service technology and IP tunneling technology to, forexample, route data packets between VMs 2024 on different hosts 2020within the data center 2000 network; note that an interior gatewayprotocol (IGP) may be used to exchange routing information within such alocal network.

In addition, a network such as the provider data center 2000 network(which is sometimes referred to as an autonomous system (AS)) may usethe mapping service technology, IP tunneling technology, and routingservice technology to route packets from the VMs 2024 to Internetdestinations, and from Internet sources to the VMs 2024. Note that anexternal gateway protocol (EGP) or border gateway protocol (BGP) istypically used for Internet routing between sources and destinations onthe Internet. FIG. 20 shows an example provider data center 2000implementing a network that provides resource virtualization technologyand that provides full Internet access via edge router(s) 2014 thatconnect to Internet transit providers, according to some embodiments.The provider data center 2000 may, for example, provide customers theability to implement virtual computing systems (VMs 2024) via a hardwarevirtualization service and the ability to implement virtualized datastores 2016 on storage resources 2018A-2018N via a storage service.

The data center 2000 network may implement IP tunneling technology,mapping service technology, and a routing service technology to routetraffic to and from virtualized resources, for example to route packetsfrom the VMs 2024 on hosts 2020 in data center 2000 to Internetdestinations, and from Internet sources to the VMs 2024. Internetsources and destinations may, for example, include computing systems2070 connected to the intermediate network 2040 and computing systems2052 connected to local networks 2050 that connect to the intermediatenetwork 2040 (e.g., via edge router(s) 2014 that connect the network2050 to Internet transit providers). The provider data center 2000network may also route packets between resources in data center 2000,for example from a VM 2024 on a host 2020 in data center 2000 to otherVMs 2024 on the same host or on other hosts 2020 in data center 2000.

A service provider that provides data center 2000 may also provideadditional data center(s) 2060 that include hardware virtualizationtechnology similar to data center 2000 and that may also be connected tointermediate network 2040. Packets may be forwarded from data center2000 to other data centers 2060, for example from a VM 2024 on a host2020 in data center 2000 to another VM on another host in another,similar data center 2060, and vice versa.

While the above describes hardware virtualization technology thatenables multiple operating systems to run concurrently on host computersas virtual machines (VMs) on the hosts, where the VMs may beinstantiated on slots on hosts that are rented or leased to customers ofthe network provider, the hardware virtualization technology may also beused to provide other computing resources, for example storage resources2018A-2018N, as virtualized resources to customers of a network providerin a similar manner.

FIG. 21 is a block diagram of an example provider network that providesa storage service and a hardware virtualization service to customers,according to some embodiments. Hardware virtualization service 2120provides multiple computation resources 2124 (e.g., VMs) to customers.The computation resources 2124 may, for example, be rented or leased tocustomers of the provider network 2100 (e.g., to a customer thatimplements customer network 2150). Each computation resource 2124 may beprovided with one or more local IP addresses. Provider network 2100 maybe configured to route packets from the local IP addresses of thecomputation resources 2124 to public Internet destinations, and frompublic Internet sources to the local IP addresses of computationresources 2124.

Provider network 2100 may provide a customer network 2150, for examplecoupled to intermediate network 2140 via local network 2156, the abilityto implement virtual computing systems 2192 via hardware virtualizationservice 2120 coupled to intermediate network 2140 and to providernetwork 2100. In some embodiments, hardware virtualization service 2120may provide one or more APIs 2102, for example a web services interface,via which a customer network 2150 may access functionality provided bythe hardware virtualization service 2120, for example via a console 2194(e.g., a web-based application, standalone application, mobileapplication, etc.). In some embodiments, at the provider network 2100,each virtual computing system 2192 at customer network 2150 maycorrespond to a computation resource 2124 that is leased, rented, orotherwise provided to customer network 2150.

From an instance of a virtual computing system 2192 and/or anothercustomer device 2190 (e.g., via console 2194), the customer may accessthe functionality of storage service 2110, for example via one or moreAPIs 2102, to access data from and store data to storage resources2118A-2118N of a virtual data store 2116 (e.g., a folder or “bucket”, avirtualized volume, a database, etc.) provided by the provider network2100. In some embodiments, a virtualized data store gateway (not shown)may be provided at the customer network 2150 that may locally cache atleast some data, for example frequently-accessed or critical data, andthat may communicate with storage service 2110 via one or morecommunications channels to upload new or modified data from a localcache so that the primary store of data (virtualized data store 2116) ismaintained. In some embodiments, a user, via a virtual computing system2192 and/or on another customer device 2190, may mount and accessvirtual data store 2116 volumes via storage service 2110 acting as astorage virtualization service, and these volumes may appear to the useras local (virtualized) storage 2198.

While not shown in FIG. 21 , the virtualization service(s) may also beaccessed from resource instances within the provider network 2100 viaAPI(s) 2102. For example, a customer, appliance service provider, orother entity may access a virtualization service from within arespective virtual network on the provider network 2100 via an API 2102to request allocation of one or more resource instances within thevirtual network or within another virtual network.

Illustrative System

In some embodiments, a system that implements a portion or all of thetechniques for image enhancement as described herein may include ageneral-purpose computer system that includes or is configured to accessone or more computer-accessible media, such as computer system 2200illustrated in FIG. 22 . In the illustrated embodiment, computer system2200 includes one or more processors 2210 coupled to a system memory2220 via an input/output (I/O) interface 2230. Computer system 2200further includes a network interface 2240 coupled to I/O interface 2230.While FIG. 22 shows computer system 2200 as a single computing device,in various embodiments a computer system 2200 may include one computingdevice or any number of computing devices configured to work together asa single computer system 2200.

In various embodiments, computer system 2200 may be a uniprocessorsystem including one processor 2210, or a multiprocessor systemincluding several processors 2210 (e.g., two, four, eight, or anothersuitable number). Processors 2210 may be any suitable processors capableof executing instructions. For example, in various embodiments,processors 2210 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, ARM, PowerPC, SPARC, or MIPS ISAs, or any othersuitable ISA. In multiprocessor systems, each of processors 2210 maycommonly, but not necessarily, implement the same ISA.

System memory 2220 may store instructions and data accessible byprocessor(s) 2210. In various embodiments, system memory 2220 may beimplemented using any suitable memory technology, such as random-accessmemory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated embodiment, program instructions and data implementing oneor more desired functions, such as those methods, techniques, and datadescribed above are shown stored within system memory 2220 as code 2225and data 2226.

In one embodiment, I/O interface 2230 may be configured to coordinateI/O traffic between processor 2210, system memory 2220, and anyperipheral devices in the device, including network interface 2240 orother peripheral interfaces. In some embodiments, I/O interface 2230 mayperform any necessary protocol, timing or other data transformations toconvert data signals from one component (e.g., system memory 2220) intoa format suitable for use by another component (e.g., processor 2210).In some embodiments, I/O interface 2230 may include support for devicesattached through various types of peripheral buses, such as a variant ofthe Peripheral Component Interconnect (PCI) bus standard or theUniversal Serial Bus (USB) standard, for example. In some embodiments,the function of I/O interface 2230 may be split into two or moreseparate components, such as a north bridge and a south bridge, forexample. Also, in some embodiments some or all of the functionality ofI/O interface 2230, such as an interface to system memory 2220, may beincorporated directly into processor 2210.

Network interface 2240 may be configured to allow data to be exchangedbetween computer system 2200 and other devices 2260 attached to anetwork or networks 2250, such as other computer systems or devices asillustrated in FIG. 1 , for example. In various embodiments, networkinterface 2240 may support communication via any suitable wired orwireless general data networks, such as types of Ethernet network, forexample. Additionally, network interface 2240 may support communicationvia telecommunications/telephony networks such as analog voice networksor digital fiber communications networks, via storage area networks(SANs) such as Fibre Channel SANs, or via I/O any other suitable type ofnetwork and/or protocol.

In some embodiments, a computer system 2200 includes one or more offloadcards 2270 (including one or more processors 2275, and possiblyincluding the one or more network interfaces 2240) that are connectedusing an I/O interface 2230 (e.g., a bus implementing a version of thePeripheral Component Interconnect—Express (PCI-E) standard, or anotherinterconnect such as a QuickPath interconnect (QPI) or UltraPathinterconnect (UPI)). For example, in some embodiments the computersystem 2200 may act as a host electronic device (e.g., operating as partof a hardware virtualization service) that hosts compute instances, andthe one or more offload cards 2270 execute a virtualization manager thatcan manage compute instances that execute on the host electronic device.As an example, in some embodiments the offload card(s) 2270 can performcompute instance management operations such as pausing and/or un-pausingcompute instances, launching and/or terminating compute instances,performing memory transfer/copying operations, etc. These managementoperations may, in some embodiments, be performed by the offload card(s)2270 in coordination with a hypervisor (e.g., upon a request from ahypervisor) that is executed by the other processors 2210A-2210N of thecomputer system 2200. However, in some embodiments the virtualizationmanager implemented by the offload card(s) 2270 can accommodate requestsfrom other entities (e.g., from compute instances themselves), and maynot coordinate with (or service) any separate hypervisor.

In some embodiments, system memory 2220 may be one embodiment of acomputer-accessible medium configured to store program instructions anddata as described above. However, in other embodiments, programinstructions and/or data may be received, sent or stored upon differenttypes of computer-accessible media. Generally speaking, acomputer-accessible medium may include non-transitory storage media ormemory media such as magnetic or optical media, e.g., disk or DVD/CDcoupled to computer system 2200 via I/O interface 2230. A non-transitorycomputer-accessible storage medium may also include any volatile ornon-volatile media such as RAM (e.g., SDRAM, double data rate (DDR)SDRAM, SRAM, etc.), read only memory (ROM), etc., that may be includedin some embodiments of computer system 2200 as system memory 2220 oranother type of memory. Further, a computer-accessible medium mayinclude transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link, such as may be implemented vianetwork interface 2240.

FIG. 23 illustrates a logical arrangement of a set of general componentsof an example computing device 2300 such as provider network 100, clientdevice(s) 102, etc. Generally, a computing device 2300 can also bereferred to as an electronic device. The techniques shown in the figuresand described herein can be implemented using code and data stored andexecuted on one or more electronic devices (e.g., a client end stationand/or server end station). Such electronic devices store andcommunicate (internally and/or with other electronic devices over anetwork) code and data using computer-readable media, such asnon-transitory computer-readable storage media (e.g., magnetic disks,optical disks, Random Access Memory (RAM), Read Only Memory (ROM), flashmemory devices, phase-change memory) and transitory computer-readablecommunication media (e.g., electrical, optical, acoustical or other formof propagated signals, such as carrier waves, infrared signals, digitalsignals). In addition, such electronic devices include hardware, such asa set of one or more processors 2302 (e.g., wherein a processor is amicroprocessor, controller, microcontroller, central processing unit,digital signal processor, application specific integrated circuit, fieldprogrammable gate array, other electronic circuitry, a combination ofone or more of the preceding) coupled to one or more other components,e.g., one or more non-transitory machine-readable storage media (e.g.,memory 2304) to store code (e.g., instructions 2314) and/or data, and aset of one or more wired or wireless network interfaces 2308 allowingthe electronic device to transmit data to and receive data from othercomputing devices, typically across one or more networks (e.g., LocalArea Networks (LANs), the Internet). The coupling of the set ofprocessors and other components is typically through one or moreinterconnects within the electronic device, (e.g., busses and possiblybridges). Thus, the non-transitory machine-readable storage media (e.g.,memory 2304) of a given electronic device typically stores code (e.g.,instructions 2314) for execution on the set of one or more processors2302 of that electronic device. One or more parts of various embodimentsmay be implemented using different combinations of software, firmware,and/or hardware.

A computing device 2300 can include some type of display element 2306,such as a touch screen or liquid crystal display (LCD), although manydevices such as portable media players might convey information viaother means, such as through audio speakers, and other types of devicessuch as server end stations may not have a display element 2306 at all.As discussed, some computing devices used in some embodiments include atleast one input and/or output component(s) 2312 able to receive inputfrom a user. This input component can include, for example, a pushbutton, touch pad, touch screen, wheel, joystick, keyboard, mouse,keypad, or any other such device or element whereby a user is able toinput a command to the device. In some embodiments, however, such adevice might be controlled through a combination of visual and/or audiocommands and utilize a microphone, camera, sensor, etc., such that auser can control the device without having to be in physical contactwith the device.

As discussed, different approaches can be implemented in variousenvironments in accordance with the described embodiments. For example,FIG. 24 illustrates an example of an environment 2400 for implementingaspects in accordance with various embodiments. For example, in someembodiments requests are HyperText Transfer Protocol (HTTP) requeststhat are received by a web server (e.g., web server 2406), and theusers, via electronic devices, may interact with the provider networkvia a web portal provided via the web server 2406 and application server2408. As will be appreciated, although a web-based environment is usedfor purposes of explanation, different environments may be used, asappropriate, to implement various embodiments. The system includes anelectronic client device 2402, which may also be referred to as a clientdevice and can be any appropriate device operable to send and receiverequests, messages or information over an appropriate network 2404 andconvey information back to a user of the device 2402. Examples of suchclient devices include personal computers (PCs), cell phones, handheldmessaging devices, laptop computers, set-top boxes, personal dataassistants, electronic book readers, wearable electronic devices (e.g.,glasses, wristbands, monitors), and the like. The one or more networks2404 can include any appropriate network, including an intranet, theInternet, a cellular network, a local area network, or any other suchnetwork or combination thereof. Components used for such a system candepend at least in part upon the type of network and/or environmentselected. Protocols and components for communicating via such a networkare well known and will not be discussed herein in detail. Communicationover the network can be enabled via wired or wireless connections andcombinations thereof. In this example, the network 2404 includes theInternet, as the environment includes a web server 2406 for receivingrequests and serving content in response thereto, although for othernetworks an alternative device serving a similar purpose could be used,as would be apparent to one of ordinary skill in the art.

The illustrative environment includes at least one application server2408 and a data store 2410. It should be understood that there can beseveral application servers, layers, or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein the term “data store” refers to any device orcombination of devices capable of storing, accessing and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server 2408 caninclude any appropriate hardware and software for integrating with thedata store 2410 as needed to execute aspects of one or more applicationsfor the client device 2402 and handling a majority of the data accessand business logic for an application. The application server 2408provides access control services in cooperation with the data store 2410and is able to generate content such as text, graphics, audio, video,etc., to be transferred to the client device 2402, which may be servedto the user by the web server in the form of HyperText Markup Language(HTML), Extensible Markup Language (XML), JavaScript Object Notation(JSON), or another appropriate unstructured or structured language inthis example. The handling of all requests and responses, as well as thedelivery of content between the client device 2402 and the applicationserver 2408, can be handled by the web server 2406. It should beunderstood that the web server 2406 and application server 2408 are notrequired and are merely example components, as structured code discussedherein can be executed on any appropriate device or host machine asdiscussed elsewhere herein.

The data store 2410 can include several separate data tables, databases,or other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing production data 2412 and user information 2416,which can be used to serve content for the production side. The datastore 2410 also is shown to include a mechanism for storing log orsession data 2414. It should be understood that there can be many otheraspects that may need to be stored in the data store, such as page imageinformation and access rights information, which can be stored in any ofthe above listed mechanisms as appropriate or in additional mechanismsin the data store 2410. The data store 2410 is operable, through logicassociated therewith, to receive instructions from the applicationserver 2408 and obtain, update, or otherwise process data in responsethereto. In one example, a user might submit a search request for acertain type of item. In this case, the data store 2410 might access theuser information 2416 to verify the identity of the user and can accessa production data 2412 to obtain information about items of that type.The information can then be returned to the user, such as in a listingof results on a web page that the user is able to view via a browser onthe user device 2402. Information for a particular item of interest canbe viewed in a dedicated page or window of the browser.

The web server 2406, application server 2408, and/or data store 2410 maybe implemented by one or more electronic devices 2420, which can also bereferred to as electronic server devices or server end stations, and mayor may not be located in different geographic locations. Each of the oneor more electronic devices 2420 may include an operating system thatprovides executable program instructions for the general administrationand operation of that device and typically will includecomputer-readable medium storing instructions that, when executed by aprocessor of the device, allow the device to perform its intendedfunctions. Suitable implementations for the operating system and generalfunctionality of the devices are known or commercially available and arereadily implemented by persons having ordinary skill in the art,particularly in light of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 24 . Thus, the depiction of the environment 2400 inFIG. 24 should be taken as being illustrative in nature and not limitingto the scope of the disclosure.

Various embodiments discussed or suggested herein can be implemented ina wide variety of operating environments, which in some cases caninclude one or more user computers, computing devices, or processingdevices which can be used to operate any of a number of applications.User or client devices can include any of a number of general-purposepersonal computers, such as desktop or laptop computers running astandard operating system, as well as cellular, wireless, and handhelddevices running mobile software and capable of supporting a number ofnetworking and messaging protocols. Such a system also can include anumber of workstations running any of a variety ofcommercially-available operating systems and other known applicationsfor purposes such as development and database management. These devicesalso can include other electronic devices, such as dummy terminals,thin-clients, gaming systems, and/or other devices capable ofcommunicating via a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TransmissionControl Protocol/Internet Protocol (TCP/IP), File Transfer Protocol(FTP), Universal Plug and Play (UPnP), Network File System (NFS), CommonInternet File System (CIFS), Extensible Messaging and Presence Protocol(XMPP), AppleTalk, etc. The network(s) can include, for example, a localarea network (LAN), a wide-area network (WAN), a virtual private network(VPN), the Internet, an intranet, an extranet, a public switchedtelephone network (PSTN), an infrared network, a wireless network, andany combination thereof.

In embodiments utilizing a web server, the web server can run any of avariety of server or mid-tier applications, including HTTP servers, FileTransfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers,data servers, Java servers, business application servers, etc. Theserver(s) also may be capable of executing programs or scripts inresponse requests from user devices, such as by executing one or moreWeb applications that may be implemented as one or more scripts orprograms written in any programming language, such as Java®, C, C# orC++, or any scripting language, such as Perl, Python, PHP, or TCL, aswell as combinations thereof. The server(s) may also include databaseservers, including without limitation those commercially available fromOracle(R), Microsoft(R), Sybase(R), IBM(R), etc. The database serversmay be relational or non-relational (e.g., “NoSQL”), distributed ornon-distributed, etc.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (SAN) familiar to those skilled inthe art. Similarly, any necessary files for performing the functionsattributed to the computers, servers, or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch screen, or keypad),and/or at least one output device (e.g., a display device, printer, orspeaker). Such a system may also include one or more storage devices,such as disk drives, optical storage devices, and solid-state storagedevices such as random-access memory (RAM) or read-only memory (ROM), aswell as removable media devices, memory cards, flash cards, etc.

Such devices also can include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device, etc.), and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium, representing remote, local, fixed, and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting, and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services, or other elementslocated within at least one working memory device, including anoperating system and application programs, such as a client applicationor web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets), or both. Further, connection to other computing devicessuch as network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules, or other data, including RAM, ROM, ElectricallyErasable Programmable Read-Only Memory (EEPROM), flash memory or othermemory technology, Compact Disc-Read Only Memory (CD-ROM), DigitalVersatile Disk (DVD) or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by a system device. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

In the preceding description, various embodiments are described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Bracketed text and blocks with dashed borders (e.g., large dashes, smalldashes, dot-dash, and dots) are used herein to illustrate optionaloperations that add additional features to some embodiments. However,such notation should not be taken to mean that these are the onlyoptions or optional operations, and/or that blocks with solid bordersare not optional in certain embodiments.

Reference numerals with suffix letters may be used to indicate thatthere can be one or multiple instances of the referenced entity invarious embodiments, and when there are multiple instances, each doesnot need to be identical but may instead share some general traits oract in common ways. Further, the particular suffixes used are not meantto imply that a particular amount of the entity exists unlessspecifically indicated to the contrary. Thus, two entities using thesame or different suffix letters may or may not have the same number ofinstances in various embodiments.

References to “one embodiment,” “an embodiment,” “an exampleembodiment,” etc., indicate that the embodiment described may include aparticular feature, structure, or characteristic, but every embodimentmay not necessarily include the particular feature, structure, orcharacteristic. Moreover, such phrases are not necessarily referring tothe same embodiment. Further, when a particular feature, structure, orcharacteristic is described in connection with an embodiment, it issubmitted that it is within the knowledge of one skilled in the art toaffect such feature, structure, or characteristic in connection withother embodiments whether or not explicitly described.

Moreover, in the various embodiments described above, unlessspecifically noted otherwise, disjunctive language such as the phrase“at least one of A, B, or C” is intended to be understood to mean eitherA, B, or C, or any combination thereof (e.g., A, B, and/or C). As such,disjunctive language is not intended to, nor should it be understood to,imply that a given embodiment requires at least one of A, at least oneof B, or at least one of C to each be present.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the disclosure asset forth in the claims.

What is claimed is:
 1. A computer-implemented method comprising:receiving a received frame of a video to be at least partially enhancedusing a trained neural network; enhancing the received frame using thetrained neural network by applying the trained neural network on thereceived frame by: generating, using a residual layer of the trainedneural network, a residual value based on a proper subset of thereceived frame and at least one corresponding image portion of apreceding frame in the video and at least one corresponding imageportion of a subsequent frame in the video, wherein a proper subset is aset less than a whole set; upscaling the received frame using upsamplingto generate an upscaled frame; and combining the upscaled frame and theresidual value to generate an enhanced frame; and outputting theenhanced frame.
 2. The computer-implemented method of claim 1, furthercomprising enhancing at least a different proper subset of the receivedframe using a different trained neural network.
 3. Thecomputer-implemented method of claim 1, further comprising performingobject recognition on the received frame to determine the proper subsetof the received frame to enhance.
 4. The computer-implemented method ofclaim 1, further comprising performing a tensor product of the upscaledframe and the received frame prior to combining the upscaled frame andthe residual value to generate the enhanced frame.
 5. Thecomputer-implemented method of claim 1, wherein the received frameincludes encoded information indicating the proper subset of thereceived frame to enhance.
 6. The computer-implemented method of claim1, further comprising receiving a request to enhance one or more framesof the video, the request including at least one of a location of thetrained neural network, a location of the video, a desired resolution,or an indication of which frames to enhance.
 7. The computer-implementedmethod of claim 1, wherein upscaling the received frame using upsamplingcomprises using bilinear upsampling.
 8. The computer-implemented methodof claim 1, wherein the trained neural network is a convolutional neuralnetwork.
 9. A computer-implemented method comprising: receiving areceived frame of a video to be at least partially enhanced using atrained neural network; enhancing the received frame using the trainedneural network by applying the trained neural network on the receivedframe by: generating, using a residual layer of the trained neuralnetwork, a residual value based on a proper subset of the received frameand at least one corresponding image portion of a higher-resolutionframe in the video, wherein a proper subset is a set less than a wholeset; upscaling the received frame using upsampling to generate anupscaled frame; and combining the upscaled frame and the residual valueto generate an enhanced frame; and outputting the enhanced frame. 10.The computer-implemented method of claim 9, further comprising enhancingat least a different proper subset of the received frame using adifferent trained neural network.
 11. The computer-implemented method ofclaim 9, further comprising performing object recognition on thereceived frame to determine the proper subset of the received frame toenhance.
 12. The computer-implemented method of claim 9, furthercomprising performing a tensor product of the upscaled frame and thereceived frame prior to combining the upscaled frame and the residualvalue to generate the enhanced frame.
 13. The computer-implementedmethod of claim 9, wherein the received frame includes encodedinformation indicating the proper subset of the received frame toenhance.
 14. The computer-implemented method of claim 9, furthercomprising receiving a request to enhance one or more frames of thevideo, the request including at least one of a location of the trainedneural network, a location of the video, a desired resolution, or anindication of which frames to enhance.
 15. The computer-implementedmethod of claim 9, wherein upscaling the received frame using upsamplingcomprises using bilinear upsampling.
 16. The computer-implemented methodof claim 9, wherein the trained neural network is a convolutional neuralnetwork.
 17. A system comprising: storage for a video; an imageenhancement service implemented by one or more electronic devices, theimage enhancement service including instructions that, upon execution,cause the image enhancement service to: receive a received frame of avideo to be at least partially enhanced using a trained neural network;enhance the received frame using the trained neural network by applyingthe trained neural network on the received frame by: generating, using aresidual layer of the trained neural network, a residual value based ona proper subset of the received frame and at least one correspondingimage portion of a preceding frame in the video and at least onecorresponding image portion of a subsequent frame in the video, whereina proper subset is a set less than a whole set; upscaling the receivedframe using upsampling to generate an upscaled frame; and combining theupscaled frame and the residual value to generate an enhanced frame; andoutput the enhanced frame.
 18. The system of claim 17, wherein the imageenhancement service includes further instructions that, upon execution,further cause the image enhancement service to enhance the receivedframe using the trained neural network by enhancing at least a differentproper subset of the received frame using a different trained neuralnetwork.
 19. The system of claim 17, wherein the image enhancementservice includes further instructions that, upon execution, furthercause the image enhancement service to enhance the received frame usingthe trained neural network by performing object recognition on thereceived frame to determine the proper subset of the received frame toenhance.
 20. The system of claim 17, wherein the image enhancementservice includes further instructions that, upon execution, furthercause the image enhancement service to enhance the received frame usingthe trained neural network by performing a tensor product of theupscaled frame and the received frame prior to combining the upscaledframe and the residual value to generate the enhanced frame.